XG Boost in Mental Health Classification

Introduction

eXtreme Gradient Boosting (XGB) algorithm

• A supervised machine learning algorithm
• Modification of Gradient Boosting Framework
• Ensemble of weak decision trees
• L1,L2 regularization
• High performance, speed, scalability
• Applications in healthcare, education, public health, finance, and engineering.

Introduction

XGBoost for mental disorder classification

• Using XGBoost on clinical and survey data
• Classification of individuals into one of four categories: Bipolar I Disorder, Bipolar II Disorder, Major Depressive Disorder, and Normal
• XGBoosts proves to be a suitable algorithm for the problem

Literature Review

eXtreme Gradient Boosting (XGBoost) in healthcare

• XGBoost strengths in real world problems handling class imbalance, heterogeneous data types, or non-linear relationships. (Chen and Guestrin 2016)
• XGBoost combined with DL for breast cancer classification with high reliability. (Liew, Hameed, and Clos 2021)
• XGBoost with biomarker data to improve depression diagnoses in a large Dutch population dataset.(Sharma and Verbeke 2020)
• XGBoost to multi-modal datasets to predict self-harm in young adults.(Xu et al. 2024)
• A hybrid algorithm (XGBoost-HOA) to classify depression, anxiety, and stress
• Dual XGBoost models applied to distinguish between deficit and non-deficit schizophrenia subtypes using fMRI features.(Zhang, Jia, and Shang 2022)
• XGBoost compared with linear regression in predicting depression among refugee children.(Saleh et al. 2024)

Literature Review

XGBoost limitations for imbalance data

• handled by optimization techniques (Zhang, Jia, and Shang 2022)
• Handled by balancing the class distribution (Sharma and Verbeke 2020)
• Preprocessing and hyperparameter tuning importance in imbalance data for accurate XGBoost algorithm.

Literature Review

XGBoost in Education

• XGBoost in academic prediction models (Hu and Song 2019)
• XGBoost out performing traditional logistic regression models in predicting learner performance (Hakkal and Ait Lahcen 2024).

XGBoost in Public health

• XGBoost used to predict daily COVID-19 cases in the United States.(Fang et al. 2022)
• A hybrid model integrating XGBoost, Random Forest, and Antlion Optimization used to predict infectious disease outbreaks.(Sivakumar and Elangovan 2023)

Literature Review

XGBoost in financial sector

• XGBoost model’s ability to rank feature importance and prevent overfitting using regularization made it a top performer in credit risk prediction tasks(Li et al. 2020)
• XGBoost to forecast volatility in the U.S. stock market, identifying the Economic Policy Uncertainty Index as a critical predictor.(Fomunyam 2023)

XGBoost In pharmaceutical research

• Application of XGBoost in drug development.(Wiens et al. 2025)

Literature Review

XGBoost in educational diagnostics

• XGBoost enhanced learner performance prediction and knowledge tracing on platforms like ASSIST09 and Algebra08. (Su et al. 2023) and (Hakkal and Ait Lahcen 2024)

XGBoost in Sports Analytics

• XGBoost used to predict ultramarathon running speeds based on demographics and environmental factors.(Nikolaidis, Knechtle, and co-authors 2023)

Methods: Extreme Gradient Boosting (XGBoost)

• Ensemble of weak decision trees
• Trained sequentially, each tree corrects previous errors
• Captures non-linear relationships within the 17-variable dataset
• Uses gradient descent to minimize loss
• Includes regularization to prevent overfitting

Methods: XGBoost Formulas

• Loss term: Measures prediction error
• Regularization term: Penalizes model complexity
• Ensures balance between accuracy and generalization

\[ \text{Obj} = \sum_{i=1}^{n} L(y_i, \hat{y}_i) + \sum_{k=1}^{K} \Omega(f_k) \]

Methods: Training the Workflow

• Stage 1: Initial Prediction
• Stage 2: Iterative Boosting (Tree k)
• Stage 3: Final Ensemble Prediction

XG Boost

Data Exploration and Visualization

• Sourced from Kaggle (2023 clinical records)
• 120 individuals assessed for mental disorders -Labels:
  • Normal (25% of total)
  • Bipolar Type I (23% of total)
  • Bipolar Type II (26% of total)
  • Depression (26% of total)
• 17 diagnostic variables
• Mix of clinical, behavioral, demographic factors

Data Exploration and Visualization

• Examined class distribution
• Identified potential imbalance or outliers
• Explored correlations between features

Cramers V

Modeling and Results

Data Preprocessing

To facilitate analysis, all categorical variables in the dataset were transformed into numerical or grouped categorical formats in Excel. The transformations were applied as follows:

• Ordinal responses (Sadness, Euphoric, Exhausted) recoded:
  • Seldom=1, Sometimes=2, Usually=3, Most-Often=4
• Binary items (Suicidal Thoughts, Mood Swing):
  • Yes=1, No=0
• 1–10 rating scales (Sexual Activity, Optimism) grouped:
  • 1–3 → Cat 1, 4–6 → Cat 2, 7–9 → Cat 3

Modeling and Results

Hyperparameter Optimization

• The dataset was randomly partitioned into an 80/20 training–testing split.
  • This split resulted in 96 patient surveys used for model training and 24 surveys reserved for independent testing.
  • The predictive target for all models was the expert clinical diagnosis
• An XGBoost model specification was created using the tidymodels framework where all major hyperparameters were set to be tuned including:
  • the number of trees
  • tree depth
  • learning rate
  • loss reduction,
  • minimum node size
  • subsampling rate
  • number of predictors considered at each split

Modeling and Results

Hyperparameter Optimization

• Five-fold stratified cross-validation was used for hyperparameter tuning.
  • Where the training dataset was partitioned into five equally sized folds
  • For each tuning iteration:
    • Models were trained on four folds
    • Validated on the remaining fold
    • Which ensured stable and unbiased performance estimates

(Singh 2024)

Modeling and Results

Hyperparameter Optimization

• A 30-point hyperparameter grid was generated using a space-filling design.
• Model performance was evaluated using Accuracy and ROC AUC.
• Performance improved with more boosting iterations and depth = 2, peaking at ~625 trees with Accuracy > 0.80 and ROC AUC > 0.90.

VisulizationTuningResults

Modeling and Results

Hyperparameter Optimization

• The select_best() function from the tidymodels framework was used to systematically extract the hyperparameter combination that achieved the highest ROC AUC across all resampled evaluations.

• The resulting optimal configuration included 3 predictors (mtry = 3), 627 boosting iterations, a minimum node size (min_n) of 2, a learning rate of 0.16501, a loss-reduction parameter of 0.0034, and a subsample proportion of 0.5655.

  • This combination represents the tuning grid entry labeled pre0_mod06_post0, which achieved the best balance between model complexity and predictive performance during cross-validation.

BestParams

Modeling and Results

Model Evaluation & Performance

Prediction

The model’s predictions on the held-out test set showed strong agreement with the expert diagnoses. Specifically:

• Bipolar Type-1: 4 of 6 cases were correctly predicted (66.7%)
  
• Bipolar Type-2: 5 of 7 cases were correctly predicted (71.4%)
  
• Depression: 6 of 7 cases were correctly predicted (85.7%)
  
• Normal: 4 of 6 cases were correctly predicted (66.7%)

Modeling and Results

Model Evaluation & Performance

• XGBoost shows moderate overall predictive performance with strong discriminative ability.
• Accuracy: 0.654 — correctly classifies ~65% of observations.
• Kappa: 0.534 — indicates moderate agreement beyond chance.
• Macro ROC AUC: 0.927 — strong ability to rank observations across all classes.
• Overall: High AUC but moderate accuracy/kappa, suggesting strong discrimination but room to improve final classification.

Modeling and Results

Feature Importance & Interepretability

• SHAP analysis shows both broadly influential and class-specific predictors across the four classes.

• Most Influential (Across Classes):

  • Mood Swing: strongest overall, especially for Bipolar I & II
    
  • Suicidal Thoughts: key for Depression and Bipolar II
    
  • Sadness: major driver for Bipolar I

• Class-Specific Influences:

  • Concentration (Cat 2): strong indicator for Normal
    
  • Authority & Respect: most impactful for Bipolar II
    
  • Ignore & Move On and Exhausted: help distinguish the Normal class

• Conclusion: Distinct SHAP patterns reveal unique psychological signatures across classes, reflecting a well-calibrated multiclass XGBoost model.

Modeling and Results

Sensitvity & Robustness Analysis

• Sensitivity analysis shows the model is stable and consistent across resampling.
  • Mean Accuracy: 0.804 — strong predictive performance.
    
  • SD Accuracy: 0.0821 — moderate variability across samples.
    
  • Mean ROC AUC: 0.957 — excellent and consistent discriminative ability.
    
  • SD ROC AUC: 0.0397 — highly stable class separation.
• Overall: High, stable AUC and low–moderate variability, indicating robust performance with slight accuracy sensitivity to data shifts.

Modeling and Results

Summary of Modeling Results

• On the 24-case test set, the model showed strong agreement with expert diagnoses and good generalization.

• It correctly identified most cases in each diagnostic category.

• Final model strengths:

  • High discriminative power: ROC AUC > 0.90 across CV and sensitivity analyses
    
  • Stable generalization: low variability across repeats
    
  • Interpretability: clear clinical insights via SHAP and feature importance
    
  • Accurate predictions: strong performance across all four diagnostic groups

• Conclusion: XGBoost is an effective, interpretable model for mental health classification in a small structured dataset, offering strong predictive value and clinically meaningful insights.

Conclusion

Study Purpose

• Evaluate effectiveness of XGBoost for predicting mental health outcomes.
• Develop, tune, and validate the model.
• Identify highest‑predictive features and assess ability to capture non‑linear relationships.
• Model showed strong accuracy and generalizability.

Conclusion

Key Findings

• Top predictors: stress indicators, behavioral patterns, wellness factors.
• High feature importance suggests relevance to mental‑health risk.
• Model effectively distinguished between classes.
• Stable performance across tuning configurations with minimal overfitting.

Implications

• XGBoost shows potential for early warning mental‑health monitoring.
• Useful for identifying at‑risk individuals based on survey data.
• Self‑reported data may introduce bias—interpret with care.

Conclusion

Future Work

• Use larger and more diverse datasets.
• Incorporate behavioral or time‑based variables.
• Compare with Random Forest, LightGBM, neural networks.
• Test in real‑world settings.
• Address fairness and ethical use of health data.

Conclusion: Overview

• Overall, XGBoost proved to be an effective and interpretable modeling approach for mental health classification in a small structured dataset, delivering both predictive value and meaningful interpretability essential for clinical research. Our XGBoost Model performed with 88% accuracy. Machine-learning approaches may provide timely insights that complement traditional assessment methods and support early identification of individuals who may benefit from targeted intervention. With further refinement and validation, predictive modeling has the potential to become an increasingly valuable component of mental-health research and practice.